Rapid Super-sampling of Multi-frame Sequences using Blind Deconvolution

Under certain conditions, multi-frame image sequences can be processed to produce images that achieve greater resolution through image registration and increased sampling. This technique, known as super-sampling, takes advantage of the spatial-temporal data available in an under-sampled imaging sequence. In this effort, the image registration is replaced by application of a fast blind deconvolution technique to remove the motion blur in the up-sampled average of the image sequence. This produces a super-sampled image with significantly decreased computational requirements compared to common methods. Method and simulated test results are presented. ocis code: 100.1830, 100,3010, 100.6640. Obtaining increased spatial information from the temporal characteristics of a multi-frame image sequence, known as super-sampling or superresolution, has received significant attention in published literature.[?, 1, 2, 3] The objective is to gain additional information from an image sequence by recovering higher spatial characteristics than can be obtained from a single image. Images produced at one sampling frequency are sampled at Research Support Instruments: Optics Letters, Vol 29(17), p. 1986, 2004. 2 a higher frequency and combined. If the conditions permit, image resolution can be improved up to the optical cutoff frequency of the imaging system. Much research has focused on the special case where frame-to-frame differences are purely translational. In these cases, the super-sampling processes follow the same general approach. The sampling frequency of each frame is increased to the desired sampling size. The translation errors of each frame are estimated through comparison with a chosen reference frame. The frames are realigned to the super-sampled grid and averaged to create the super-sampled image. The process of determining the translation errors is a major challenge and can be computationally intensive. Here we present an alternative approach. As before, each frame is upsampled. Then, without realignment, the frames are averaged together. This produces a single image that has complicated motion blur embedded in it. This motion blur is removed using a blind deconvolution algorithm. Blind deconvolution techniques can also be computationally intensive and require significant user input. However, a recent innovation, the Selfdeconvolving Data Reconstruction Algorithm (SeDDaRA), allows fast and effective identification of an unknown blur function enabling restoration of the image.[4, 5, 6] Thus, it is possible to produce a super-sampled image without measuring frame-to-frame translation errors, and in a timely fashion. To demonstrate the super-sampling/blind deconvolution technique, an imaging simulation was produced that generates not only frame-toframe motion, but also motion blur within each frame. The in-frame motion blur is important since it occurs in most imaging sequences and is not removed with image registration algorithms. However, blind deconvolution techniques can remove at least some of this type of blur in addition to the frame-to-frame motion blur. A multi-frame sequence was created by applying 256 random translations, calculated using a first-order autoregressive expression, to a still image. Each segment of sixteen images were averaged to form a sixteen frame video sequence with motion blur and random noise. As described, the frames are up-sampled and averaged together. The averaged image contains a complicated motion blur that can be removed using SeDDaRA. A mathematical representation of the averaged image in Research Support Instruments: Optics Letters, Vol 29(17), p. 1986, 2004. 3 frequency space G(u, v) is G(u, v) = F (u, v) D(u, v) +W (u, v) (1) where (u, v) are the coordinates in frequency space, F (u, v) is the real scene,D(u, v) represents the blur function, andW (u, v) is a noise term. For this study, a pseudo-inverse filter was used to filter the influence of the blur from the image. The deconvolution is given by F (u, v) ≈ G(u, v) D(u, v) |D(u, v)|2 + C2 (2) where the parameter C2 is typically chosen by trial and error. This filter is fast, easy to apply, and provides a good approximation of the Wiener filter. [7, 8, 9, 10] Without explicit knowledge of D(u, v), the function must be estimated through blind deconvolution. Research on blind deconvolution extends back several decades, but studies by Ayers and Dainty [11] spurred an increase in activity in the astronomical community [12]. All methods require some prior knowledge of either the scene, [13] the scene statistics, [14, 15] or the shape of the blur function. [16] Most of these techniques are iterative and can only be applied within certain restraints. In contrast, the SeDDaRA approach can be applied to all images provided a suitable representation of the desired spatial frequency can be found. [4] The SeDDaRA process assumes the transfer function has the form D(u, v) = [KGS{|G(u, v)−W (u, v)|}] α(u,v) (3) where α(u, v) is a tuning parameter andKG is a real, positive scalar chosen to ensure |D(u, v)| ≤ 1. Application of the smoothing filter S{...} assumes that D(u, v) is a slowly varying function. Assumptions for this calculation are explained in a previous paper. [4] After some derivation, α(u, v) is found to be α(u, v) ≈ Ln[KGS|G(u, v)−W (u, v)|]− Ln[KF ′S{|F (u, v)|}] Ln[KGS|G(u, v)−W (u, v)|] (4) where F (u, v) is an alternate truth image that satisfies KF ′S{|F ′(u, v)|} ≈ KFS{|F (u, v)|}. (5) Research Support Instruments: Optics Letters, Vol 29(17), p. 1986, 2004. 4 The presence of a smoothing filter greatly relaxes this condition. In Equation 4,KG andKF ′ must be determined such that |D(u, v)| ≤ 1. This condition is satisfied if we setKG = 1/Max[S{|G(u, v)|}] andKF ′ = 1/Max[S{|F (u, v)|}]. Once D(u, v) has been extracted from the averaged image, both functions are inserted into Equation 2 to remove the blur. Figure 1 (left) shows a portion of the 600×600 truth image used for the simulation. The maximum allowable translation of 3.0 pixels produced an average frame-to frame motion of 1.2. The frames were down-sampled to 256× 256 using bi-cubic interpolation and rotated 90 degrees. The frames were then up-sampled to a 512 × 512 size. The frames were averaged, as shown in Figure 1 (right), and processed using blind deconvolution, Figure 2. There is significant evidence of super-sampling in the image. The image is sharper, and small objects possess greater contrast against their backgrounds. Examination of the images at a finer scale supports this claim. Figure 3 displays a portion of the image alongside the canal for each step in the process. The original, non-degraded image is at location (a). The (b) image is a single frame at the lower sampling frequency, i.e. the image produced by the simulated camera. The familiar ‘stair step’ pattern is evident. The (c) image is the average of all frames at the final sampling frequency. The (d) image is the fully processed result showing a sharper edge than the averaged image and less of a stair step pattern along the edge. The other features closely match their respective shapes in the truth image. The final image has artifacts produced by the interpolation process that manifest themselves as ringing from edges and lines not parallel to image edges. Since these anomalies are not spatially invariant, the deconvolution amplifies these artifacts. These artifacts are not expected to occur in a real, non-simulated sequence. For comparison, the image sequence was super-sampled by using a phase correlation method tomeasure the translation errors. The re-aligned images were then averaged together. A portion of the result is shown in Figure 4. The averaged image (b) shows obvious blur when compared to the super-sample/blind deconvolution method (a). Application of an edge filter (c) removes some blur but shows less evidence of super-sampling. Application of blind deconvolution to image (b) shows a result (d) that is quite similar to image (a), but requires significantly more computations. Thismethod of super-sampling presents a distinct advantage over other Research Support Instruments: Optics Letters, Vol 29(17), p. 1986, 2004. 5 Figure 1: (Left) The left side of the original or ‘truth’ image used in the simulation. This is an aerial view of Norfolk, Virginia, taken by the Ikonos satellite. (Right) The right side of the average of sixteen translated frames up-sampled to the final sampling frequency. Research Support Instruments: Optics Letters, Vol 29(17), p. 1986, 2004. 6 Figure 2: The fully processed image produce by applying a blind deconvolution to the averaged image. Research Support Instruments: Optics Letters, Vol 29(17), p. 1986, 2004. 7